This notebook will include informal meta-analyses of different metrics and methods for evaluating surgical skill.
The reported metrics compare differences between novices and expert surgeons.
It is informal because it’s not based on systematic review, and because some studies have been included with very relaxed conditions. For example, I have picked the novices and experts without comparing their definitions between studies. Novice = weakest skill group in the study, expert = strongest skill group in the study. If a study included more than 2 groups, I picked the weakest (=novice) and strongest (=expert) groups’ results and discarded the others. If a study included more than 1 task, or several sub-tasks, I picked the one with largest difference between groups.
Many papers did report means and standard deviations explicitly, so they had to be estimated from boxplots/barplots, or by some other means
For example, sometimes studies reported only mean or median, but no SE/SD. I estimated the SD/SE in those cases based e.g. on the SD of some other similar metric that they reported, or the SD of previous results for the same metric. See the excel file for notes on each study.
May or may not be turned into more systematic meta-analysis later.
Example metrics that will be most likely included (Bolded ones have priority)
Full list of papers and metrics can be found in the excel file shared in the repo:
Last update: 20.6.2022.
If you notice errors or know some good studies to be included, feel free to forward them to
jani.koskinen [ at ] uef.fi
or use the form below TBD
These values are used as input in the R meta package’s metagen function.
For more information, check:
Forest plot explanation
How many samples needed at some effect size d? At alpha = 0.05 and power = 0.8 and using t-test. Assuming independent trials (e.g. no multiple measurements from same participants etc.)
Hover mouse over the points in the plot to see the values. Some baseline effect sizes from the meta-analyses given as baseline:
IT = Idle Time TT = Task Time BD = Bimanual Dexterity TEPR = Task-Evoked Pupil Reaction/Dilation (Esimated without one outlier study removed)
Task time is the time taken to complete a task. Task can be short like a single knot or some longer complex task.
Load data
df.time <- read_excel('data/surgical_metrics.xlsx', sheet='task_time')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Koskinen et al. | 2022 | Utilizing Grasp Monitoring to Predict Microsurgical Expertise | Journal of Surgical Research |
| Chainey et al. | 2021 | Eye-Hand Coordination of Neurosurgeons: Evidence of Action-Related Fixation in Microsuturing | World Neurosurgery |
| Harada et al. | 2015 | Assessing microneurosurgical skill with medico-engineering technology | World Neurosurgery |
| Vedula et al. | 2016 | Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment | Journal of Surgical Education |
| Judkins et al. | 2009 | Objective evaluation of expert and novice performance during robotic surgical training tasks | Surgical Endoscopy |
| Smith et al. | 2002 | Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy | Surgical Endoscopy and Other Interventional Techniques |
| Francis et al. | 2002 | The performance of master surgeons on the Advanced Dundee Endoscopic Psychomotor Tester: Contrast validity study | Archives of Surgery |
| Moorthy et al. | 2004 | Bimodal assessment of laparoscopic suturing skills: Construct and concurrent validity | Surgical Endoscopy and Other Interventional Techniques |
| Van Sickle et al. | 2008 | Construct validity of an objective assessment method for laparoscopic intracorporeal suturing and knot tying | The American Journal of Surgery |
| Xeroulis et al. | 2009 | Simulation in laparoscopic surgery: A concurrent validity study for FLS | Surgical Endoscopy and Other Interventional Techniques |
| Huffman et al. | 2020 | Optimizing Assessment of Surgical Knot Tying Skill | Journal of Surgical Education |
| Law et al. | 2004 | Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment | Proceedings of the Eye tracking research & applications symposium on Eye tracking research & applications - ETRA’2004 |
| Kazemi et al. | 2010 | Assessing suturing techniques using a virtual reality surgical simulator | Microsurgery |
| O’Toole et al. | 1999 | Measuring and Developing Suturing Technique with a Virtual Reality Surgical Simulator | Journal ofthe American College of Surgeons |
| Zheng et al. | 2021 | Action-related eye measures to assess surgical expertise | BJS Open |
| Datta et al. | 2001 | The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model | Journal of the American College of Surgeons |
| Pagador et al. | 2012 | Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment | International Journal of Computer Assisted Radiology and Surgery |
| Aggarwal et al. | 2007 | An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room | Annals of Surgery |
| Wilson et al. | 2010 | Psychomotor control in a virtual laparoscopic surgery training environment: Gaze control parameters differentiate novices from experts | Surgical Endoscopy |
| Hofstad et al. | 2013 | A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance | Surgical Endoscopy and Other Interventional Techniques |
| Hung et al. | 2018 | Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study | Journal of Urology |
| Yamaguchi et al. | 2011 | Objective assessment of laparoscopic suturing skills using a motion-tracking system | Surgical Endoscopy |
| Pellen et al. | 2009 | Laparoscopic surgical skills assessment: Can simulators replace experts? | World Journal of Surgery |
| Pastewski et al. | 2021 | Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills | Journal of Surgical Education |
| Chmarra et al. | 2010 | Objective classification of residents based on their psychomotor laparoscopic skills | Surgical Endoscopy and Other Interventional Techniques |
m.time <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.time,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Time to completion in Surgery")
summary(m.time)
## Review: Time to completion in Surgery
##
## SMD 95%-CI %W(random)
## Koskinen et al. 1.8413 [ 1.4135; 2.2691] 4.5
## Chainey et al. 0.7034 [ 0.0383; 1.3686] 4.4
## Harada et al. 1.5503 [ 0.8551; 2.2456] 4.4
## Vedula et al. 2.2149 [ 1.7299; 2.6999] 4.5
## Judkins et al. 5.3971 [ 3.8216; 6.9726] 3.7
## Smith et al. 8.0559 [ 5.5551; 10.5568] 2.9
## Francis et al. 0.9801 [ 0.3227; 1.6375] 4.4
## Moorthy et al. 1.4157 [ 0.3397; 2.4917] 4.1
## Van Sickle et al. 2.1365 [ 1.0202; 3.2528] 4.1
## Xeroulis et al. 2.5525 [ 1.2650; 3.8400] 4.0
## Huffman et al. 6.5116 [ 4.9174; 8.1059] 3.7
## Law et al. 2.0257 [ 1.3401; 2.7112] 4.4
## Kazemi et al. 0.8354 [-0.3084; 1.9791] 4.1
## O'Toole et al. 1.7086 [ 0.6569; 2.7602] 4.1
## Zheng et al. 1.9382 [ 0.6894; 3.1871] 4.0
## Datta et al. 2.1791 [ 1.1762; 3.1819] 4.2
## Pagador et al. 6.3695 [ 2.5221; 10.2170] 2.0
## Aggarwal et al. 0.1873 [-0.4390; 0.8136] 4.4
## Wilson et al. 1.3349 [ 0.1520; 2.5179] 4.1
## Hofstad et al. 1.3791 [ 0.3199; 2.4382] 4.1
## Hung et al. 2.2342 [ 1.7203; 2.7481] 4.5
## Yamaguchi et al. 4.5870 [ 2.7625; 6.4116] 3.5
## Pellen et al. 5.6362 [ 3.6128; 7.6596] 3.3
## Pastewski et al. 0.5600 [-0.1171; 1.2371] 4.4
## Chmarra et al. 0.9810 [ 0.0706; 1.8914] 4.2
##
## Number of studies combined: k = 25
##
## SMD 95%-CI t p-value
## Random effects model 2.3748 [1.5627; 3.1869] 6.04 < 0.0001
##
## Quantifying heterogeneity:
## tau^2 = 2.9577 [1.8111; 7.7816]; tau = 1.7198 [1.3458; 2.7895]
## I^2 = 86.4% [81.1%; 90.2%]; H = 2.71 [2.30; 3.19]
##
## Test of heterogeneity:
## Q d.f. p-value
## 176.04 24 < 0.0001
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.time,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Time to completion in Surgery")
#dev.print(pdf, "figures/forest_time.pdf", width=10, height=10)
Time to completion is by far the most often reported metric. It is often reported even when it is not the main focus of the study.
Bimanual dexterity is a measure of how well the surgeon is able to use both hands at the same time. Note that there are many different ways for calculating “ability to use both hands simultaneously.”
Load data
df.biman <- read_excel('data/surgical_metrics.xlsx', sheet='tool_bimanual')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Koskinen et al. | 2022 | Movement-level process modeling of microsurgical bimanual and unimanual tasks | International Journal of Computer Assisted Radiology and Surgery |
| Hofstad et al. | 2017 | Psychomotor skills assessment by motion analysis in minimally invasive surgery on an animal organ | Minimally Invasive Therapy and Allied Technologies |
| Demirel et al. | 2022 | Scoring metrics for assessing skills in arthroscopic rotator cuff repair: performance comparison study of novice and expert surgeons | International Journal of Computer Assisted Radiology and Surgery |
| Islam et al. | 2016 | Affordable, web-based surgical skill training and evaluation tool | Journal of Biomedical Informatics |
| Zulbaran-Rojas et al. | 2021 | Utilization of Flexible-Wearable Sensors to Describe the Kinematics of Surgical Proficiency | Journal of Surgical Research |
| Mori et al. | 2022 | Validation of a novel virtual reality simulation system with the focus on training for surgical dissection during laparoscopic sigmoid colectomy | BMC Surgery |
Run meta-analysis
m.biman <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.biman,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Bimanual dexterity in Surgery")
Print results
summary(m.biman)
## Review: Bimanual dexterity in Surgery
##
## SMD 95%-CI %W(random)
## Koskinen et al. -3.0589 [ -3.8825; -2.2353] 17.4
## Hofstad et al. -3.0127 [ -4.6473; -1.3782] 16.0
## Demirel et al. -2.0314 [ -3.0378; -1.0251] 17.1
## Islam et al. -8.6969 [-10.7900; -6.6039] 15.1
## Zulbaran-Rojas et al. -0.8250 [ -1.7586; 0.1085] 17.2
## Mori et al. -2.6867 [ -3.6936; -1.6799] 17.1
##
## Number of studies combined: k = 6
##
## SMD 95%-CI t p-value
## Random effects model -3.2752 [-6.0448; -0.5057] -3.04 0.0288
##
## Quantifying heterogeneity:
## tau^2 = 6.0648 [2.0109; 43.9963]; tau = 2.4627 [1.4181; 6.6330]
## I^2 = 89.8% [80.5%; 94.7%]; H = 3.13 [2.27; 4.33]
##
## Test of heterogeneity:
## Q d.f. p-value
## 49.11 5 < 0.0001
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.biman,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Bimanual dexterity in Surgery")
#dev.print(pdf, "figures/forest_biman.pdf", width=8, height=8)
Analysis of bimanual dexterity is made harder because there are so many different definitions for it.
Number of tool movements made during the task. Note: I have included here the grasp results from our paper (and other studies that analyzed only one type of action/movement)
Load data
df.toolmvt <- read_excel('data/surgical_metrics.xlsx', sheet='tool_movements')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Datta et al. | 2001 | The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model | Journal of the American College of Surgeons |
| Pagador et al. | 2012 | Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment | International Journal of Computer Assisted Radiology and Surgery |
| Koskinen et al. | 2022 | Utilizing Grasp Monitoring to Predict Microsurgical Expertise | Journal of Surgical Research |
| Bann et al. | 2003 | Measurement of surgical dexterity using motion analysis of simple bench tasks | World Journal of Surgery |
| Smith et al. | 2002 | Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy | Surgical Endoscopy and Other Interventional Techniques |
| Aggarwal et al. | 2007 | An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room | Annals of Surgery |
| Yamaguchi et al. | 2007 | Construct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulator | Surgical Endoscopy and Other Interventional Techniques |
| Goldbraikh et al. | 2021 | Video-based fully automatic assessment of open surgery suturing skills | International Journal of Computer Assisted Radiology and Surgery |
| Vedula et al. | 2016 | Task-Level vs . Segment-Level Quantitative Metrics for Surgical Skill Assessment | Journal of Surgical Education |
| Wilson et al. | 2010 | Psychomotor control in a virtual laparoscopic surgery training environment: Gaze control parameters differentiate novices from experts | Surgical Endoscopy |
| Hofstad et al. | 2013 | A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance | Surgical Endoscopy and Other Interventional Techniques |
Run meta-analysis
m.toolmvt <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.toolmvt,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Tool movements in Surgery")
summary(m.toolmvt)
## Review: Tool movements in Surgery
##
## SMD 95%-CI %W(random)
## Datta et al. 2.0390 [ 1.0607; 3.0174] 9.6
## Pagador et al. 10.0866 [ 4.2364; 15.9368] 3.7
## Koskinen et al. 1.3393 [ 0.7781; 1.9006] 10.0
## Bann et al. 1.2504 [ 0.4629; 2.0380] 9.8
## Smith et al. 5.9403 [ 4.0136; 7.8671] 8.5
## Aggarwal et al. -0.0641 [-0.6894; 0.5612] 9.9
## Yamaguchi et al. 2.3074 [ 1.3887; 3.2260] 9.7
## Goldbraikh et al. 2.6143 [ 1.8535; 3.3751] 9.8
## Vedula et al. 6.5233 [ 5.6418; 7.4047] 9.7
## Wilson et al. 0.5955 [-0.4889; 1.6799] 9.5
## Hofstad et al. 0.9586 [-0.0444; 1.9617] 9.6
##
## Number of studies combined: k = 11
##
## SMD 95%-CI t p-value
## Random effects model 2.5911 [0.8299; 4.3523] 3.28 0.0083
##
## Quantifying heterogeneity:
## tau^2 = 5.1496 [2.4728; 26.0028]; tau = 2.2693 [1.5725; 5.0993]
## I^2 = 94.7% [92.2%; 96.4%]; H = 4.33 [3.58; 5.24]
##
## Test of heterogeneity:
## Q d.f. p-value
## 187.46 10 < 0.0001
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.toolmvt,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool movements in Surgery")
#dev.print(pdf, "figures/forest_toolmvt.pdf", width=8, height=8)
Tool movements are perhaps the second most often reported metric. Different papers measure, analyze and report them differently. Often connected to “movement efficiency”.
Tool idle time measures how long the tools were not being used, either as time or as fraction of the complete task time.
Load data
df.toolidle <- read_excel('data/surgical_metrics.xlsx', sheet='tool_idle')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Koskinen et al. | 2021 | Movement-level process modeling of microsurgical bimanual and unimanual tasks | International Journal of Computer Assisted Radiology and Surgery |
| Uemura et al. | 2015 | Procedural surgical skill assessment in laparoscopic training environments | International Journal of Computer Assisted Radiology and Surgery |
| D’Angelo et al. | 2015 | Idle time: An underdeveloped performance metric for assessing surgical skill | American Journal of Surgery |
| Mackenzie et al. | 2021 | Enhanced Training Benefits of Video Recording Surgery With Automated Hand Motion Analysis | World Journal of Surgery |
| Oropesa et al. | 2013 | Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery | Surgical Innovation |
| Hung et al. | 2018 | Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study | Journal of Urology |
| Topalli et al. | 2018 | Eye-Hand Coordination Patterns of Intermediate and Novice Surgeons in a Simulation-Based Endoscopic Surgery Training Environment | Journal of Eye Movement Research |
Run meta-analysis
m.toolidle <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.toolidle,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Idle time in Surgery")
summary(m.toolidle)
## Review: Idle time in Surgery
##
## SMD 95%-CI %W(random)
## Koskinen et al. 2.5600 [ 1.8069; 3.3131] 17.7
## Uemura et al. 1.6933 [ 0.7816; 2.6050] 16.2
## D'Angelo et al. 2.7724 [ 1.0363; 4.5085] 9.3
## Mackenzie et al. 0.3642 [-1.6449; 2.3734] 7.7
## Oropesa et al. 0.9556 [-0.1828; 2.0941] 14.0
## Hung et al. 0.6990 [ 0.2837; 1.1144] 20.8
## Topalli et al. 0.6159 [-0.4828; 1.7147] 14.3
##
## Number of studies combined: k = 7
##
## SMD 95%-CI t p-value
## Random effects model 1.3803 [0.5279; 2.2326] 3.96 0.0074
##
## Quantifying heterogeneity:
## tau^2 = 0.5520 [0.0915; 4.0820]; tau = 0.7430 [0.3024; 2.0204]
## I^2 = 75.3% [47.7%; 88.3%]; H = 2.01 [1.38; 2.93]
##
## Test of heterogeneity:
## Q d.f. p-value
## 24.29 6 0.0005
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.toolidle,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Idle time in Surgery")
#dev.print(pdf, "figures/forest_toolidle.pdf", width=8, height=8)
Not many papers that focused on idle time.
How much the tools travel during the task.
Load data
df.toolpl <- read_excel('data/surgical_metrics.xlsx', sheet='tool_path_length')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Aggarwal et al. | 2007 | An evaluation of the feasibility, validity, and reliability of laparoscopic skills assessment in the operating room | Annals of Surgery |
| Moorthy et al. | 2004 | Bimodal assessment of laparoscopic suturing skills: Construct and concurrent validity | Surgical Endoscopy and Other Interventional Techniques |
| Smith et al. | 2002 | Motion analysis: A tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy | Surgical Endoscopy and Other Interventional Techniques |
| Pagador et al. | 2012 | Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment | International Journal of Computer Assisted Radiology and Surgery |
| Goldbraikh et al. | 2021 | Video-based fully automatic assessment of open surgery suturing skills | International Journal of Computer Assisted Radiology and Surgery |
| Jimbo et al. | 2017 | A new innovative laparoscopic fundoplication training simulator with a surgical skill validation system | Surgical Endoscopy |
| Hofstad et al. | 2013 | A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance | Surgical Endoscopy and Other Interventional Techniques |
| Oropesa et al. | 2013 | Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery | Surgical Innovation |
| Pellen et al. | 2009 | Laparoscopic surgical skills assessment: Can simulators replace experts? | World Journal of Surgery |
| D’Angelo et al. | 2015 | Idle time: An underdeveloped performance metric for assessing surgical skill | American Journal of Surgery |
| Hung et al. | 2018 | Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study | Journal of Urology |
| Vedula et al. | 2016 | Task-Level vs . Segment-Level Quantitative Metrics for Surgical Skill Assessment | Journal of Surgical Education |
| Yamaguchi et al. | 2011 | Objective assessment of laparoscopic suturing skills using a motion-tracking system | Surgical Endoscopy |
| Harada et al. | 2015 | Assessing Microneurosurgical Skill with Medico-Engineering Technology | World Neurosurgery |
| Ebina et al. | 2021 | Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs | Surgical Endoscopy |
| Chmarra et al. | 2010 | Objective classification of residents based on their psychomotor laparoscopic skills | Surgical Endoscopy and Other Interventional Techniques |
Run meta-analysis
m.toolpl <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.toolpl,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Tool path length in Surgery")
summary(m.toolpl)
## Review: Tool path length in Surgery
##
## SMD 95%-CI %W(random)
## Aggarwal et al. 0.0647 [-0.5606; 0.6900] 7.6
## Moorthy et al. 1.3161 [ 0.2542; 2.3780] 6.0
## Smith et al. 2.6541 [ 1.5194; 3.7889] 5.7
## Pagador et al. 6.2865 [ 2.4827; 10.0904] 1.2
## Goldbraikh et al. 2.0174 [ 1.3325; 2.7024] 7.4
## Jimbo et al. 0.8695 [ 0.1950; 1.5441] 7.5
## Hofstad et al. 0.7989 [-0.1876; 1.7853] 6.3
## Oropesa et al. 0.2889 [-0.8108; 1.3885] 5.8
## Pellen et al. 2.0960 [ 0.9877; 3.2042] 5.8
## D'Angelo et al. 1.7506 [ 0.3193; 3.1819] 4.7
## Hung et al. 1.8000 [ 1.3220; 2.2779] 8.1
## Vedula et al. 2.4204 [ 1.9215; 2.9194] 8.1
## Yamaguchi et al. 3.3661 [ 1.8874; 4.8449] 4.5
## Harada et al. 1.0214 [ 0.3743; 1.6685] 7.6
## Ebina et al. 0.9071 [ 0.1861; 1.6281] 7.3
## Chmarra et al. 1.2076 [ 0.2706; 2.1447] 6.4
##
## Number of studies combined: k = 16
##
## SMD 95%-CI t p-value
## Random effects model 1.5109 [0.9657; 2.0561] 5.91 < 0.0001
##
## Quantifying heterogeneity:
## tau^2 = 0.5801 [0.2782; 3.3711]; tau = 0.7617 [0.5275; 1.8360]
## I^2 = 78.4% [65.5%; 86.5%]; H = 2.15 [1.70; 2.72]
##
## Test of heterogeneity:
## Q d.f. p-value
## 69.48 15 < 0.0001
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.toolpl,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool path length in Surgery")
#dev.print(pdf, "figures/forest_toolpl.pdf", width=8, height=8)
Tool path length also a very common metric. Most studies report that novices have much larger path length, indicating less effective movements. Results differ based on task and surgical
Tool velocity/speed measures how fast the surgical tool or tools are moving.
Load data
df.toolvelocity <- read_excel('data/surgical_metrics.xlsx', sheet='tool_velocity')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Davids et al. | 2021 | Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation. | World Neurosurgery |
| Pastewski et al. | 2021 | Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills | Journal of Surgical Education |
| Hwang et al. | 2006 | Correlating motor performance with surgical error in laparoscopic cholecystectomy | Surgical Endoscopy and Other Interventional Techniques |
| Ebina et al. | 2021 | Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs | Surgical Endoscopy |
| Jimbo et al. | 2017 | A new innovative laparoscopic fundoplication training simulator with a surgical skill validation system | Surgical Endoscopy |
| Judkins et al. | 2009 | Objective evaluation of expert and novice performance during robotic surgical training tasks | Surgical Endoscopy |
| Hofstad et al. | 2013 | A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance | Surgical Endoscopy and Other Interventional Techniques |
| Frasier et al. | 2016 | A marker-less technique for measuring kinematics in the operating room | Surgery (United States) |
| Azari et al. | 2018 | Can surgical performance for varying experience be measured from hand motions? | Proceedings of the Human Factors and Ergonomics Society |
| Pagador et al. | 2012 | Decomposition and analysis of laparoscopic suturing task using tool-motion analysis (TMA): Improving the objective assessment | International Journal of Computer Assisted Radiology and Surgery |
Run meta-analysis
m.toolvelocity <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.toolvelocity,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Tool velocity in Surgery")
summary(m.toolvelocity)
## Review: Tool velocity in Surgery
##
## SMD 95%-CI %W(random)
## Davids et al. 0.5140 [-1.5370; 2.5650] 5.0
## Pastewski et al. -0.7177 [-1.4028; -0.0326] 12.8
## Hwang et al. 6.1176 [ 1.5045; 10.7307] 1.3
## Ebina et al. -0.8684 [-1.5865; -0.1503] 12.5
## Jimbo et al. -0.7654 [-1.4334; -0.0974] 12.9
## Judkins et al. 0.6675 [-0.0690; 1.4039] 12.4
## Hofstad et al. 1.0086 [-0.0002; 2.0174] 10.4
## Frasier et al. -1.1447 [-1.7143; -0.5751] 13.6
## Azari et al. -0.2982 [-1.2042; 0.6078] 11.1
## Pagador et al. 0.0585 [-1.3278; 1.4448] 7.9
##
## Number of studies combined: k = 10
##
## SMD 95%-CI t p-value
## Random effects model -0.1917 [-0.9642; 0.5809] -0.56 0.5883
##
## Quantifying heterogeneity:
## tau^2 = 0.4917 [0.1990; 10.7136]; tau = 0.7012 [0.4461; 3.2732]
## I^2 = 74.0% [51.2%; 86.1%]; H = 1.96 [1.43; 2.68]
##
## Test of heterogeneity:
## Q d.f. p-value
## 34.58 9 < 0.0001
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.toolvelocity,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool velocity in Surgery")
#dev.print(pdf, "figures/forest_toolvelocity.pdf", width=8, height=8)
Velocity (and related metrics like acceleration) are semi-popular method. Results seem to vary a lot, sometimes novices are faster and sometimes experts are faster. May depend on task?
Tool acceleration measures how much the tool/tools accelerate during the task.
Load data
df.toolacc <- read_excel('data/surgical_metrics.xlsx', sheet='tool_acceleration')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Azari et al. | 2018 | Can surgical performance for varying experience be measured from hand motions? | Proceedings of the Human Factors and Ergonomics Society |
| Frasier et al. | 2016 | A marker-less technique for measuring kinematics in the operating room | Surgery (United States) |
| Ebina et al. | 2021 | Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs | Surgical Endoscopy |
| Pastewski et al. | 2021 | Analysis of Instrument Motion and the Impact of Residency Level and Concurrent Distraction on Laparoscopic Skills | Journal of Surgical Education |
| Davids et al. | 2021 | Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation. | World Neurosurgery |
Run meta-analysis
m.toolacc <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.toolacc,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Tool acceleration in Surgery")
summary(m.toolacc)
## Review: Tool acceleration in Surgery
##
## SMD 95%-CI %W(random)
## Azari et al. -0.3713 [-1.2803; 0.5377] 18.5
## Frasier et al. -1.0298 [-1.5922; -0.4674] 27.7
## Ebina et al. -0.7891 [-1.5016; -0.0767] 23.3
## Pastewski et al. 0.1911 [-0.4748; 0.8570] 24.7
## Davids et al. -0.0233 [-2.0633; 2.0167] 5.8
##
## Number of studies combined: k = 5
##
## SMD 95%-CI t p-value
## Random effects model -0.4926 [-1.1602; 0.1750] -2.05 0.1099
##
## Quantifying heterogeneity:
## tau^2 = 0.1828 [0.0000; 1.9388]; tau = 0.4275 [0.0000; 1.3924]
## I^2 = 52.3% [0.0%; 82.5%]; H = 1.45 [1.00; 2.39]
##
## Test of heterogeneity:
## Q d.f. p-value
## 8.39 4 0.0782
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.toolacc,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Tool acceleration in Surgery")
#dev.print(pdf, "figures/forest_toolacceleration.pdf", width=8, height=8)
Not many papers that focused on tool accelerations. Jerk (third derivative of position, derivative of acceleration) is much more popular.
Jerk is the third derivative of the surgical instruments position, and measures how smooth the movements are.
Load data
df.jerk <- read_excel('data/surgical_metrics.xlsx', sheet='tool_jerk')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Ghasemloonia et al. | 2017 | Surgical Skill Assessment Using Motion Quality and Smoothness | Journal of Surgical Education |
| Hwang et al. | 2006 | Correlating motor performance with surgical error in laparoscopic cholecystectomy | Surgical Endoscopy and Other Interventional Techniques |
| Ebina et al. | 2021 | Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs | Surgical Endoscopy |
| Azari et al. | 2018 | Can surgical performance for varying experience be measured from hand motions? | Proceedings of the Human Factors and Ergonomics Society |
| Davids et al. | 2021 | Automated vision-based microsurgical skill analysis in neurosurgery using deep learning: Development and preclinical validation. | World Neurosurgery |
| Oropesa et al. | 2013 | Relevance of Motion-Related Assessment Metrics in Laparoscopic Surgery | Surgical Innovation |
| Maithel et al | 2005 | Simulated laparoscopy using a head-mounted display vs traditional video monitor: An assessment of performance and muscle fatigue | Surgical Endoscopy and Other Interventional Techniques |
| Liang et al. | 2018 | Motion control skill assessment based on kinematic analysis of robotic end-effector movements | The International Journal of Medical Robotics and Computer Assisted Surgery |
| Islam et al. | 2016 | Affordable, web-based surgical skill training and evaluation tool | Journal of Biomedical Informatics |
| Hofstad et al. | 2017 | Psychomotor skills assessment by motion analysis in minimally invasive surgery on an animal organ | Minimally Invasive Therapy and Allied Technologies |
| Shafiel et al. | 2017 | Motor Skill Evaluation During Robot-Assisted Surgery | Volume 5A: 41st Mechanisms and Robotics Conference |
| Chmarra et al. | 2010 | Objective classification of residents based on their psychomotor laparoscopic skills | Surgical Endoscopy and Other Interventional Techniques |
Run meta-analysis
m.jerk <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.jerk,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Jerk in Surgery")
summary(m.jerk)
## Review: Jerk in Surgery
##
## SMD 95%-CI %W(random)
## Ghasemloonia et al. 1.7090 [ 1.1677; 2.2504] 9.6
## Hwang et al. 2.6183 [ 0.1709; 5.0658] 5.0
## Ebina et al. -0.9365 [-1.6598; -0.2133] 9.3
## Azari et al. -0.1972 [-0.8043; 0.4098] 9.5
## Davids et al. 0.1307 [-1.9101; 2.1714] 5.9
## Oropesa et al. -0.9775 [-2.1179; 0.1629] 8.3
## Maithel et al 1.6060 [ 0.7461; 2.4658] 9.0
## Liang et al. 0.1596 [-0.7184; 1.0377] 8.9
## Islam et al. 3.6094 [ 2.4907; 4.7280] 8.3
## Hofstad et al. 1.3201 [-0.1549; 2.7952] 7.4
## Shafiel et al. 0.4174 [ 0.2951; 0.5397] 10.1
## Chmarra et al. 0.8995 [-0.0026; 1.8015] 8.9
##
## Number of studies combined: k = 12
##
## SMD 95%-CI t p-value
## Random effects model 0.7932 [-0.0699; 1.6564] 2.02 0.0681
##
## Quantifying heterogeneity:
## tau^2 = 1.5140 [0.6120; 5.0878]; tau = 1.2305 [0.7823; 2.2556]
## I^2 = 87.6% [80.2%; 92.2%]; H = 2.84 [2.25; 3.59]
##
## Test of heterogeneity:
## Q d.f. p-value
## 88.85 11 < 0.0001
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.jerk,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Jerk in Surgery")
#dev.print(pdf, "figures/forest_tooljerk.pdf", width=8, height=8)
TBD
Tool force is the force the surgeon uses when they e.g. grasp something using the surgical tools.
Load data
df.force <- read_excel('data/surgical_metrics.xlsx', sheet='tool_force')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Harada et al. | 2015 | Assessing Microneurosurgical Skill with Medico-Engineering Technology | World Neurosurgery |
| Prasad et al. | 2016 | Objective Assessment of Laparoscopic Force and Psychomotor Skills in a Novel Virtual Reality-Based Haptic Simulator | Journal of Surgical Education |
| Horeman et al. | 2014 | Assessment of Laparoscopic Skills Based on Force and Motion Parameters | IEEE Transactions on Biomedical Engineering |
| Trejos et al. | 2014 | Development of force-based metrics for skills assessment in minimally invasive surgery | Surgical Endoscopy |
| Woodrow et al. | 2007 | Training and evaluating spinal surgeons: The development of novel performance measures | Spine |
| Sugiyama et al. | 2018 | Forces of Tool-Tissue Interaction to Assess Surgical Skill Level | JAMA Surgery |
Run meta-analysis
m.force <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.force,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Force use in Surgery")
summary(m.force)
## Review: Force use in Surgery
##
## SMD 95%-CI %W(random)
## Harada et al. 0.5357 [-0.0830; 1.1545] 19.3
## Prasad et al. 1.2450 [ 0.6378; 1.8523] 19.3
## Horeman et al. 2.7082 [ 1.5555; 3.8609] 16.3
## Trejos et al. 1.5351 [ 0.2224; 2.8477] 15.3
## Woodrow et al. 3.8205 [ 2.2438; 5.3972] 13.7
## Sugiyama et al. 0.3071 [-0.8880; 1.5022] 16.0
##
## Number of studies combined: k = 6
##
## SMD 95%-CI t p-value
## Random effects model 1.5936 [0.2394; 2.9477] 3.03 0.0292
##
## Quantifying heterogeneity:
## tau^2 = 1.2415 [0.2843; 10.4203]; tau = 1.1142 [0.5332; 3.2280]
## I^2 = 79.2% [54.5%; 90.5%]; H = 2.19 [1.48; 3.24]
##
## Test of heterogeneity:
## Q d.f. p-value
## 24.02 5 0.0002
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.force,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Force use in Surgery")
#dev.print(pdf, "figures/forest_toolforce.pdf", width=8, height=8)
Forces analyzed somewhat commonly, but often not between novices and experts, but within tasks, or tools, or skill groups.
Pupil size measures cognitive workload, stress, and million other things.
Load data
df.pupil <- read_excel('data/surgical_metrics.xlsx', sheet='pupil_dilation')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Castner et al. | 2020 | Pupil diameter differentiates expertise in dental radiography visual search | PLOS ONE |
| Cabrera-Mino et al. | 2019 | Task-Evoked Pupillary Responses in Nursing Simulation as an Indicator of Stress and Cognitive Load | Clinical Simulation in Nursing |
| Bednarik et al. | 2018 | Pupil Size As an Indicator of Visual-motor Workload and Expertise in Microsurgical Training Tasks | Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications |
| Gunawardena et al. | 2019 | Assessing Surgeons’ Skill Level in Laparoscopic Cholecystectomy using Eye Metrics | Eye Tracking Research and Applications Symposium (ETRA) |
| Dilley et al. | 2020 | Visual behaviour in robotic surgery—Demonstrating the validity of the simulated environment | International Journal of Medical Robotics and Computer Assisted Surgery |
| Gao et al. | 2018 | Quantitative evaluations of the effects of noise on mental workloads based on pupil dilation during laparoscopic surgery | American Surgeon |
Run meta-analysis
m.pupil <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.pupil,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Pupil dilation in Surgery")
summary(m.pupil)
## Review: Pupil dilation in Surgery
##
## SMD 95%-CI %W(random)
## Castner et al. 0.7877 [ 0.6671; 0.9083] 17.6
## Cabrera-Mino et al. 0.8255 [ 0.0502; 1.6009] 16.7
## Bednarik et al. -2.9791 [-3.5250; -2.4332] 17.1
## Gunawardena et al. 1.5927 [ 0.3701; 2.8152] 15.4
## Dilley et al. -0.0152 [-0.7136; 0.6833] 16.8
## Gao et al. 1.2184 [ 0.3422; 2.0946] 16.4
##
## Number of studies combined: k = 6
##
## SMD 95%-CI t p-value
## Random effects model 0.2082 [-1.5479; 1.9643] 0.30 0.7728
##
## Quantifying heterogeneity:
## tau^2 = 2.6787 [0.9588; 16.5443]; tau = 1.6367 [0.9792; 4.0675]
## I^2 = 97.3% [95.7%; 98.2%]; H = 6.03 [4.85; 7.51]
##
## Test of heterogeneity:
## Q d.f. p-value
## 182.07 5 < 0.0001
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Q-profile method for confidence interval of tau^2 and tau
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.pupil,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Pupil dilation in Surgery")
#dev.print(pdf, "figures/forest_pupil.pdf", width=8, height=8)
Prior research indicates that higher stress/cognitive workload -> larger pupil size. This is seen in most studies. In Bednarik et al. (2018), the effect is reversed. For that study, I picked needle piercing segment (because it was quaranteed to have un-interrupted visual contact from the participant). It can be that experts focused more on this, and had larger cognitive workload and pupil dilations.
Not that many studies that have measured pupil dilations and compared surgical novices and experts directly. Some used measures like ICA or Entropy (not included here). Pupil dilations used in other fields more often.
Grasps are the number of times the surgeon had to grasp something using surgical instruments
Load data
df.grasp <- read_excel('data/surgical_metrics.xlsx', sheet='tool_grasps')
Print studies
| Author | Year | Study | Journal |
|---|---|---|---|
| Koskinen et al. | 2022 | Utilizing Grasp Monitoring to Predict Microsurgical Expertise | Journal of Surgical Research |
| Cao et al. | 1996 | Task and Motion Analysis in Endoscopic Surgery | 5th Annual Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems |
Run meta-analysis
m.grasp <- metagen(TE=g,
seTE=SDg,
studlab=Author,
data=df.grasp,
sm="SMD",
fixed=FALSE,
random=TRUE,
method.tau="REML",
hakn=TRUE,
title="Grasping in Surgery")
summary(m.grasp)
## Review: Grasping in Surgery
##
## SMD 95%-CI %W(random)
## Koskinen et al. 1.3393 [ 0.7781; 1.9006] 94.2
## Cao et al. 1.0000 [-1.2605; 3.2605] 5.8
##
## Number of studies combined: k = 2
##
## SMD 95%-CI t p-value
## Random effects model 1.3196 [0.3112; 2.3280] 16.63 0.0382
##
## Quantifying heterogeneity:
## tau^2 = 0; tau = 0; I^2 = 0.0%; H = 1.00
##
## Test of heterogeneity:
## Q d.f. p-value
## 0.08 1 0.7752
##
## Details on meta-analytical method:
## - Inverse variance method
## - Restricted maximum-likelihood estimator for tau^2
## - Hartung-Knapp adjustment for random effects model
Plot forest
forest.meta(m.grasp,sortvar=g, prediction=TRUE, prin.tau2=TRUE, title="Grasping in Surgery")
#dev.print(pdf, "figures/forest_grasp.pdf", width=8, height=8)
Not many papers that focused on grasps. I have included thus the grasp results also to the “Tool movements” analysis.